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TITLE OF THE INVENTION 
IMAGE SEARCH APPARATUS AND METHOD, AND COMPUTER READABLE 

MEMORY 

5 BACKGROUND OF THE INVENTION 

The present invention relates to an image search 
apparatus and method for searching an image database 
that stores a plurality of image data for desired image 

10 data, and a computer readable memory. 

Various image search apparatuses for searching 
databases which store a large number of image data for 
desired image data have been proposed. The search 
methods used in these image search apparatuses are 

15 roughly classified into two methods; 

• a method of storing non-image information such as 
keywords, photographing dates, and the like in 
association with image data, and conducting a search 
based on such information; and 

20 - a method of conducting a search on the basis of 

the image feature amounts (liominance/color difference 
information, image frequency, histogram, and the like) 
of image data itself. 

In the latter method, a method of submitting 

25 certain image data, and searching image data using the 
image feature amounts of that image data as query keys 
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is called similar image search. This method can 
effectively provide a search interface which is friendly 
to a user who has no special knowledge about image 
processing. 

5 However, when a handwritten illustration is used 

as a query criteria in a similar image search, a desired 
image cannot be obtained unless an appropriate 
illustration is drawn, thus requiring drawing skills or 
troublesome works in drawing. Also, a desired image 

10 cannot be obtained unless colors used in drawing are 
appropriate. Furthermore, the color to be used is 
normally designated on the basis of numerical values 
such as R, G, and B luminance values, or the like. 
However, it is not easy for the user who has no 

15 knowledge about the image processing to immediately 

understood such values, thus disabling efficient search 
operation. 

SUMMARY OF THE INVENTION 
20 The present invention has been made in 

consideration of the aforementioned problems, and has as 
its object to provide an image search apparatus and 
method, which can quickly reflect user's will, and can 
efficiently make an image search, and a computer 
25 readable memory. 
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In order to achieve the above object, an image 
search apparatus according to the present invention 
comprises the following arrangement. 

That iS;. an image search apparatus for searching 
5 an image database that stores a plurality of image data 
for desired image data, comprises: 

storage means for storing the plurality of image 
data in correspondence with image feature amounts of the 
image data; 

10 input means for inputting an image using an input 

windows- 
image feature amount computing means for computing 
an image feature amount of the image input by the input 
means; 

15 image similarity computing means for computing 

image similarity on the basis of the image feature 
amount computed by the image feature amount computing 
means, and the image feature amounts of the image data 
stored in the storage means; 

20 image display means for displaying a list of image 

data as search results on the basis of the image 
similarity computed by the image similarity computing 
means; and 

read-out means for reading out the image data 
25 stored in the storage means into the input window. 
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In order to achieve the above object, an image 
search method according to the present invention 
comprises the following arrangement. 

That is, an image search method for searching an 
5 image database that stores a plurality of image data for 
desired image data, comprises: 

the storage step of storing the plurality of image 
data in a storage medium in correspondence with image 
feature amounts of the image datab- 
le the image feature amount computing step of 

computing an image feature amount of an image input on 
an input window; 

the image similarity computing step of computing 
image similarity on the basis of the image feature 
15 amount computed in the image feature amount computing 
step, and the image feature amounts of the image data 
stored in the storage medium in the storage step; 

the image display step of displaying a list of 
image data as search results on the basis of the image 
20 similarity computed in the image similarity computing 
step; and 

the read-out step of reading out the image data 
stored in the storage step into the input window. 

In order to achieve the above object, a computer 
25 readable memory according to the present invention 
comprises the following arrangement. 
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That is, a computer readable memory that stores a 
program code of an image search for searching an image 
database that stores a plurality of image data for 
desired image data, comprises: 
5 a program code of the storage step of storing the 

plurality of image data in a storage medium in 
correspondence with image feature amounts of the image 
data; 

a program code of the image feature amount 
10 computing step of computing an image feature amount of 
an image input on an input window; 

a program code of the image similarity computing 
step of computing image similarity on the basis of the 
image feature amount computed in the image feature 
15 amount computing step, and the image feature amounts of 
the image data stored in the storage medium in the 
storage step; 

a program code of the image display step of 
displaying a list of image data as search results on the 
20 basis of the image similarity computed in the image 
similarity computing step; and 

a program code of the read-out step of reading out 
the image data stored in the storage step into the input 
window . 
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In order to achieve the above object, an image 
search apparatus according to the present invention 
comprises the following arrangement. 

That is, an image search apparatus for searching 
an image database that stores a plurality of image data 
for desired image data, comprises: 

storage means for storing the plurality of image 
data in correspondence with image feature amounts of the 
image data; 

input means for inputting an image using an input 

windows- 
image feature amount computing means for computing 

an image feature amount of the image input by the input 

means; 

image similarity computing means for computing 
image similarity on the basis of the image feature 
amount computed by the image feature amount computing 
means, and the image feature amounts of the image data 
stored in the storage means; 

image display means for displaying a list of image 
data as search results on the basis of the image 
similarity computed by the image similarity computing 
means; and 

designation means for designating a color used in 
the image to be drawn by the input means on the basis of 
the image data displayed by the image display means. 



In order to achieve the above object^ an image 
search method according to the present invention 
comprises the following arrangement- 

That is, an image search method for searching an 
image database that stores a plurality of image data for 
desired image data, comprises: 

the storage step of storing the plurality of image 
data in a storage medium in correspondence with image 
feature amounts of the image data; 

the image feature amount computing step of 
computing an image feature amount of an image input on 
an input window; 

the image similarity computing step of computing 
image similarity on the basis of the image feature 
amount computed in the image feature amount computing 
step, and the image feature amounts of the image data 
stored in the storage medium in the storage step; 

the image display step of displaying a list of 
image data as search results on the basis of the image 
similarity computed in the image similarity computing 
step; and 

the designation step of designating a color used 
in the image to be drawn on the input window on the 
basis of the image data displayed in the image display 
step. 



In order to achieve the above object, a computer 
readable memory according to the present invention 
comprises the following arrangement. 

That is, a computer readable memory that stores a 
5 program code of an image search for searching an image 
database that stores a plurality of image data for 
desired image data, comprises: 

a program code of the storage step of storing the 
plurality of image data in a storage medium in 
10 correspondence with image feature amounts of the image 
data; 

a program code of the image feature amount 
computing step of computing an image feature amount of 
an image input on an input window; 

1^ ^ program code of the image similarity computing 

step of computing image similarity on the basis of the 
image feature amount computed in the image feature 
amount computing step, and the image feature amounts of 
the image data stored in the storage medium in the 

20 storage step; 

a program code of the image display step of 
displaying a list of image data as search results on the 
basis of the image similarity computed in the image 
similarity computing step; and 

25 a program code of the designation step of 

designating a color used in the image to be drawn on the 
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input window on the basis of the image data displayed ir 
the image display step. 

In order to achieve the above object, an image 
search apparatus according to the present invention 
5 comprises the following arrangement. 

That is, an image search apparatus for searching 
an image database that stores a plurality of image data 
for desired image data, comprises: 

storage means for storing the plurality of image 
10 data in correspondence with image feature amounts of the 
image data; 

selection means for selecting one of a plurality 
of different input methods; 

input means for inputting an image using the input 
15 method selected by the selection means; 

image feature amount computing means for computing 
an image feature amount of the image input by the input 
means ; 

image similarity computing means for computing 
20 image similarity on the basis of the image feature 

amount computed by the image feature amount computing 
means, and the image feature amounts of the image data 
stored in the storage means; and 

image display means for displaying a list of image 
25 data as search results on the basis of the image 
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similarity computed by the image similarity computing 
means . 

In order to achieve the above object, an image 
search method according to the present invention 
comprises the following arrangement. 

That is, an image search method for searching an 
image database that stores a plurality of image data for 
desired image data, comprises: 

the storage step of storing the plurality of image 
data in a storage medium in correspondence with image 
feature amounts of the image data; 

the selection step of selecting one of a plurality 
of different input methods; 

the input step of inputting an image using the 
input method selected in the selection step; 

the image feature amount computing step of 
computing an image feature amount of an image input in 
the input step; 

the image similarity computing step of computing 
image similarity on the basis of the image feature 
amount computed in the image feature amount computing 
step, and the image feature amounts of the image data 
stored in the storage medium in the storage step; and 

the image display step of displaying a list of 
image data as search results on the basis of the image 



similarity computed in the image similarity computing 
step. 

In order to achieve the above object, a computer 
readable memory according to the present invention 
5 comprises the following arrangement. 

That is, a computer readable memory that stores a 
program code of an image search for searching an image 
database that stores a plurality of image data for 
desired image data, comprises: 
10 a program code of the storage step of storing the 

plurality of image data in a storage medium in 
correspondence with image feature amounts of the image 
data; 

a program code of the selection step of selecting 
15 one of a plurality of different input methods; 

a program code of the input step of inputting an 
image using the input method selected in the selection 
step; 

a program code of the image feature amount 
20 computing step of computing an image feature amount of 
an image input in the input step; 

a program code of the image similarity computing 
step of computing image similarity on the basis of the 
image feature amount computed in the image feature 
25 amount computing step, and the image feature amounts of 
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the image data stored in the storage medium in the 
storage step; and 

a program code of the image display step of 
displaying a list of image data as search results on the 
basis of the image similarity computed in the image 
similarity computing step. 

Other features and advantages of the present 
invention will be apparent from the following description 
taken in conjunction with the accompanying drawings, in 
which like reference characters designate the same or 
similar parts throughout the figures thereof, 

BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1 is a block diagram showing the arrangement 

of an image search apparatus according to the first 

embodiment of the present inventions- 
Fig. 2 shows a control window displayed on a 

display unit upon executing a search process of the 

first embodiments- 
Fig. 3 is a flow chart showing an outline of the 

processes executed by the image search apparatus of the 

first embodiment; 

Fig. 4 is a flow chart showing the details of the 

process executed in step S191 in the first embodiment; 
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Fig, 5 is a view for explaining image segmentation 
for extracting an image feature amount in the first 
embodiment; 

Fig. 6 is a flow chart showing the computation 
5 process of the image feature amount in the first 
embodiment; 

Fig, 7 is a flow chart showing the details of the 
method of computing the R, G, and B average values in 
the first embodiment; 
10 Fig. 8 is a flow chart showing the computation 

process of an image distance S(n) in the first 
embodiment ; 

Fig. 9 is a flow chart showing the details of the 
similar image search process in the first embodiment; 
15 and 

Fig. 10 is a flow chart showing an outline of the 
processes executed by an image search apparatus of the 
second embodiment. 

20 DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The preferred embodiments of the present invention 
will be described in detail hereinafter with reference 
to the accompanying drawings . 
<First Embodiment> 
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Fig. 1 is a block diagram showing the arrangement 
of an image search apparatus according to the first 
embodiment . 

Referring to Fig. 1, reference numeral 101 denotes 
5 a CPU for controlling the entire system. Reference 
numeral 102 denotes a keyboard; and 102a, a pointing 
device (mouse) . The keyboard 102 is used together with 
the mouse for inputting data into the system and drawing 
an illustration which is used as a query criteria for a 

10 similar image search. Reference numeral 103 denotes a 
display unit which comprises a CRT, liquid crystal 
display, or the like, and displays a user interface used 
for drawing an image serving as a query criteria, and 
image data as a search result, and the like. Reference 

15 numeral 104 denotes a ROM; and 105, a RAM. These ROM and 
RAM construct a memory device of the system, and store 
programs executed by the system and data used by the 
system. Reference numeral 106 denotes a hard disk 
device; and 107, a floppy disk device. The hard disk 

20 device and floppy disk device construct an external 

storage device used as a file system of the system. The 
hard disk device 106 stores a plurality of image data to 
be searched. Reference numeral 108 denotes a printer for 
recording an image or the like displayed on the display 

25 unit 103 on a recording medium. 
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The control window displayed on the display unit 
103 upon executing the search process of the first 
embodiment will be explained below with reference to 
Fig. 2, 

5 Fig. 2 shows the control window displayed on the 

display unit upon executing the search process in the 
first embodiment , 

Reference numeral 21 denotes a user drawing area; 
22, a cursor; 23, color selection scroll bars; 24, a 

10 clear button; 25, a search button; 27, a next candidate 
display button; and 28, a processing end button. Also, 
reference numerals 26a to 26h denote areas for 
displaying icon images corresponding to image data as 
search results. Reference numeral 29 denotes a random 

15 button. 

The user can draw an illustration, which is 
similar to the image wanted and used as a query criteria, 
on the user drawing area 21 using the drawing tools 
implemented by software. The software process upon 

20 drawing an illustration will be briefly described below. 

The color selection scroll bars 23 are used for 
selecting a pen color used in drawing, and designate R, 
G, and B values in turn from the uppermost one. Upon 
pressing the clear button 24, the entire user drawing 

25 area 21 is painted in white. The user moves the cursor 
22 using the pointing device 102a and can draw a free 
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curve on the user drawing area 21. Upon pressing the 
processing end button 28, the control window is closed, 
and the processing ends. Upon pressing the random button 
29, image data stored in the hard disk device 106 are 
5 randomly selected, and icon images corresponding to the 
selected image data are displayed on the areas 26a to 
26h. 

An outline of the processes executed by the image 
search apparatus of the first embodiment will be 

10 explained below with reference to Fig. 3, 

Fig, 3 is a flow chart showing an outline of the 
processes executed by the image search apparatus of the 
first embodiment. 

In step S191, the user draws an illustration, 

15 which is similar to the image data wanted and stored on 
the hard disk device 106, and is used as a query 
criteria, on the display unit 103. In step S192, the 
image feature amount of the drawn illustration is 
computed. In step S193, a similar image search is made 

20 on the basis of the computed image feature amount. In 
step S194, similar images (icon images) found by search 
are displayed on the display unit 103. It is then 
checked in step S195 if the user has clicked a given 
displayed icon image using the pointing device 102a. If 

25 the user has clicked the icon image (YES in step S195) , 
the flow advances to step S196. On the other hand, if 
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the user has not clicked any icon image (NO in step 
S195) , the flow advances to step S197. 

In step S196, the icon image selected in step S195 
is copied onto the user drawing area 21, and the flow 
5 returns to step S192. 

It is checked in step S197 if search operation is 
to end. If search operation is not to end (NO in step 
S197) , the flow returns to step S191. On the other hand, 
if search operation is to end (YES in step S197) , the 

10 processing ends. 

With the above-mentioned processes, if the list of 
icon images displayed as the search results contains an 
image which is close to a desired image, the user need 
only click that icon image, and the selected icon image 

15 is copied onto the user drawing area 21. The user can 
modify the copied image to generate a new illustration 
(image) used as a query criteria, and can make a similar 
image search again. At the instance of clicking that 
illustration (image) , a similar image search is 

20 automatically started using the illustration (image) as 
a query criteria . 

The details of the processes executed in the 
individual steps will be described below, 
{Description of Step S191} 

25 In step S191, the user draws an illustration, 

which is used as a query criteria, on the user drawing 
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area 21, and the flow advances to the next step S192 at 
an appropriate timing. The process at that time will be 
explained below with reference to Fig. 4. At that timing, 
the system monitors the movement of the mouse 102a, and 
5 every time a movement is monitored, the image feature 
amount of the image drawn so far is computed. 

Fig. 4 is a flow chart showing the details of the 
process executed in step S191 in the first embodiment. 

Note that xO and yO are variables for storing the 
10 previous position of the cursor 22, and xl and yl are 

variables that store the current position of the cursor 
22. 

It is checked in step S31 if the user has moved 
the mouse 102a. If the user has not moved the mouse yet 

15 (NO in step S31) , the flow returns to step S31. That is, 
the process in this step forms a loop for monitoring the 
movement of the mouse 102a. On the other hand, if the 
user has moved the mouse (YES in step S31) , the flow 
advances to step S32. 

20 It is checked in step S32 if the mouse button of 

the mouse 102a is being held down. If the mouse button 
is not held down (NO in step S32) , the flow advances to 
step S34 to substitute the current position (xl, yl) of 
the cursor 22 in (xO, yO), and the flow returns to step 

25 S31. In this manner, the cursor 22 alone can be moved 
without drawing any stroke. 
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On the other hand, if the mouse button is being 
held down (YES in step S32) , i.e., if the user is 
dragging the mouse, the flow advances to step S33. In 
step S33, a line is drawn between the previous position 
5 (xO, yO) and the current position (xl, yl) of the cursor 
22 in a color determined by the color selection scroll 
bars 23. 

In step S35, the current position (xl, yl) of the 
cursor 22 is substituted in (xO, yO) to end step S191, 

10 and the flow advances to step S192. 

This process makes the user feel as if a search 
were automatically executed every time he or she adds a 
stroke to the illustration. 
{Description of Step S192} 

15 In step S192, the image feature amount of the 

illustration drawn in step S191 is computed. 

As shown in Fig. 5, the size of the user drawing 
area 21 in the first embodiment is defined by W pixels 
(horizontal) x H pixels (vertical) . This area is 

20 segmented into a total of six sub-areas, i.e., 3 

(horizontal) x 2 (vertical) sub-areas (0, 0), (1, 0),,.,, 
(2, 1) in turn from the upper left sub-area. The R, G, 
and B average values of these sub-areas are computed, 
and a total of 18 numerical values are used as the image 

25 feature amount of the drawn illustration. 
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The computation process of the image feature 
amount will be described below with reference to Fig, 6, 

Fig. 6 is a flow chart showing the computation 
process of the image feature amount in the first 
5 embodiment . 

In step 351 f a variable k is initialized to zero. 
In step S52, a variable j is initialized to zero. In 
step S53, a variable i is initialized to zero. In step 
S54, the R average value of a sub-area (i, j) is 

10 substituted in the k-th element d(k) of a matrix d. Also^ 
the G and B average values are respectively substituted 
in d{k+l) and d(k+2) . Note that the method of computing 
the R, and B average values will be described in 
detail later with the aid of the flow chart in Fig. 7. 

15 In step S55, k is incremented by "3". In step S56, 

i is incremented by "1". In step S57, i is compared with 
"2". If i > 2 (YES in step S57), the flow advances to 
step S58. On the other hand, if i < 2 (NO in step 351), 
the flow returns to step S54. 

20 In step S58, the variable j is incremented by "1", 

In step S59, the variable j is compared with "1". If j > 
1 (YES in step S59) , the process ends. On the other hand, 
if j < 1 (NO in step S59) , the flow returns to step S53. 
Upon completion of the process, the computed 

25 values of the image feature amount of the drawn 

illustration are stored in the matrix d() having 18 
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elements. Note that the drawn illustration (image) is 
segmented into six rectangular areas having equal areas 
to compute image feature amount values in this 
embodiment. However, the shape of the segmented area is 
5 not limited to a rectangle but may be other complicated 
shapes, and the number of segmented areas may be 
increased/decreased. When the number of segmented areas 
is increased/decreased, the number of elements of the 
image feature amount is not 18 but increases/decreases 

10 accordingly. 

The method of computing the R, G, and B average 
values in step S54 in Fig. 6 will be described in detail 
below using Fig. 7. 

Fig. 7 is a flow chart showing the details of the 

15 method of computing the R, G, and B average values in 
the first embodiment. 

Assume that image data of the drawn illustration 
is stored in three matrices R(X, Y) , G(X, Y) , and B(X, 
Y) . Note that 0 < X < W and 0 < Y < H, and the start 

20 point (0, 0) is set at the upper left corner of the 

image. In the following flow, the R, G, and B average 
values of an area within the range of XO < X < XI and YO 
< Y < Yl are calculated, and are respectively returned 
to variables DR, DG, and DB. Furthermore, since an area 

25 corresponding to the sub-area (i, j) in step S192 
corresponds to: 
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XO = XI - W*(i + l)/3 

YO - H^j/2 Yl - H*(j + l)/2 
the flow chart is executed after constants XO, XI, YO, 
and Yl are initialized, as described above. 
5 In step S61, the variables DR, DG, and DB are 

initialized to zero. In step S62, a variable Y is 
initialized to YO, In step S63, a variable X is 
initialized to XO. In step S64, R(X, Y) is added to the 
variable DR. Similarly, G(X, Y) and B(X, Y) are 

10 respectively added to the variables DG and DB. 

In step S65, the variable X is incremented by "1". 
In step S66, the variable X is compared with XI. If X = 
XI (YES in step S66) , the flow advances to step S67. On 
the other hand, if X XI (NO in step 366), the flow 

15 returns to step S64. 

In step S67, the variable Y is incremented by "1", 
In step S68, the variable Y is compared with Yl. If Y ^ 
Yl (YES in step S68), the flow advances to step S69. On 
the other hand, if Y # Yl (NO in step S68), the flow 

20 returns to step S63. In step S69, the variables DR, DG, 
and DB are respectively divided by (XI - X0)^(Y1 - YO) . 
This divisor indicates the number of pixels in the area. 
That is, the variables DR, DG, and DB indicate the 
average densities obtained by dividing the sum totals of 

25 pixel densities in the area by the number of pixels. 
{Description of Step S193} 
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In step S193, the similar image search is made on 
the basis of the image feature amount computed in step 
S192. 

The hard disk device 106 stores N image data, and 

5 their image feature amounts are computed by the 

aforementioned process in advance and stored. Image data 

may be stored in a standard file format such as JPEG, 

FlashPix, or the like, which is known to those skilled 

in the art, or may be stored in a file format unique to 

10 a so-called RDBMS (relational database management 

system) . Assume that the image feature amounts are 

stored in a two-dimensional matrix D{n, i) having a size 

N*18 (for 0 < n < N, 0 < i < 18) . 

At this time, an image distance S{n) between the 

15 drawn illustration (image data) and n-th image data 

stored in the hard disk device 106 is defined by: 
S(n)=X(D(n,i)-d(iF 

I 

As this image distance S (n) is smaller, it is 
determined that image data has higher similarity. 

20 Initially, the image distances S (n) (for 0 < n < 

N) between all the N image data stored in the hard disk 
device 10 6 and the drawn illustration (image data) are 
computed. Next, a similar image search is made by 
selecting M (0 < M < N) image data in ascending order of 

25 image distance S (n) . The computation process of the 
image distance S(n), and the similar image search 
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process for selecting M image data will be respectively 
described below using Figs. 8 and 9. 

Fig. 8 is a flow chart showing the computation 
process of the image distance S(n) in the first 
5 embodiment . 

In step S71, variables min and n are initialized 
to zero, and a variable L is initialized to a 
sufficiently large value. In step S72, variables i and 
S(n) are initialized to zero. In step S73, a square of 
10 the difference between D(n, i) and d{i) is added to S (n) . 
In step S74, the variable i is incremented by "1". 

In step S75, the variable i is compared with "18". 
If i = 18 (YES in step S75) , the flow advances to step 
S76. On the other hand, if i ^ 18 (NO in step S75) , the 
15 flow returns to step S73. 

In step S76, the variable n is incremented by "1". 
In step S77, the variable n is compared with N. If n = N 
(YES in step Sll) , the processing ends. On the other 
hand, if n N (NO in step S77) , the flow returns to 
20 step S72. 

Upon completion of the process, the image 
distances S (n) between the drawn illustration (image 
data) and all the image data stored in the hard disk 
device 106 are stored in a matrix S (n) . A similar image 
25 search process for selecting M image data in ascending 
order of image distance S (n) and storing numbers 
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corresponding to the selected image order in a matrix 
T{) will be described in detail below using Fig. 9. 

Fig, 9 is a flow chart showing the details of the 
similar image search process in the first embodiment. 
5 In step S81, a variable j is initialized to zero. 

In step S82, a variable i is initialized to zero. In 
step S83, a variable min is initialized to zero and a 
variable L is initialized to a sufficiently large value. 
In step S84, S(i) is compared with L. If S(i) < L (YES 

10 in step S84), the flow advances to step S85. On the 
other hand, if S(i) > L (NO in step S84), the flow 
advances to step S8 6. 

In step S85, the value i is substituted in the 
variable min, and S(i) is substituted in L. In step S86, 

15 the variable i is incremented by "1". In step S87, i is 
compared with N. If i = N (YES in step S87), the flow 
advances to step S88. On the other hand, if i N (NO in 
step S87), the flow returns to step S84, 

In step S88, the value min is substituted in T(j). 

20 In step S89, a sufficiently large value is substituted 
in S(min), In step S810, the variable j is incremented 
by "1". In step S811, the variable j is compared with M. 
If j = M (YES in step S811), the processing ends. On the 
other hand, if j 7^ M (NO in step S811) , the flow returns 

25 to step S82. 
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Upon completion of the process, the image numbers 
of image data stored in the hard disk device 10 6 are 
stored in the matrix T(j) (for 0 < j < M) in descending 
order of similarity with the drawn illustration (image 
5 data) . 

{Description of Step S194} 

The processing contents will be explained below 
with reference to the control window shown in Fig. 2. 
The areas 26a to 26h respectively display icon 
10 images obtained by displaying similar images found by 
the search process in a reduced scale. The area 2 6a 
displays an image corresponding to T(0) with highest 
similarity, the area 26b displays an image corresponding 
to T(l),..., and the area 26h displays an image with 
15 lowest similarity among these images. 

Note that the reduced-scale display may be 
implemented by decoding image data stored in the hard 
disk device 106 and displaying the decoded image on the 
screen in a reduced scale. When image data has 
20 low-resolution icon data for an icon like FlashPix as a 
standard image format, a reduced-scale image may be 
displayed using that icon data. 

Each icon image can be "selected" using the 
pointing device 102a- Upon pressing the next candidate 
25 button 27, next candidates, i.e., icon images of image 
data corresponding to T(8) to T(15), are displayed on 
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the areas 26a to 26h, This operation can be repeated 
until T(]yi-1) is reached. 
{Description of Step S195} 

It is checked in step S195 if the user has 
selected one of the displayed icon images. If the user 
has not selected any image, the flow advances to step 
S197, On the other hand, if the user has selected one 
icon image, the flow advances to step S196. 
{Description of Step 8196} 

In step S196, the icon image selected by the user 
is copied onto the user drawing area 21, In this case, 
image data displayed on the display unit 103 may be 
copied or image data used for displaying a list of image 
data in step S194 may be used. In this manner, when an 
icon image close to a desired image appears as a search 
result, the user need only click that icon image to copy 
it onto the user drawing area 21, and can modify the 
contents of the copied image. 

Fig. 2 illustrates the user drawing area 21 to 
have substantially the same size as those of the areas 
26a to 26h for displaying icon images. However, the 
present invention is not limited to such specific layout. 
For example, the user drawing area 21 may have a 
relatively larger size to make drawing of an 
illustration easier, and the areas 26a to 26h may have a 
smaller size. When the clicked icon image is copied onto 
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the user drawing area 21, the icon image data to be 
copied is enlarged to just fall within the user drawing 
area 21. 

The flow then returns to step S192 to make a 
5 similar image search using the icon image copied onto 
the user drawing area 21 as a query criteria, 
{Description of Step S197} 

It is checked in step S197 if the processing end 
button 28 has been pressed. If the button 28 has been 
10 pressed, the processing ends. On the other hand, if the 
button 28 has not been pressed, the flow returns to step 
S191, and the user can continue to draw the illustration. 

As described above, according to the first 
embodiment, the user can select image data stored in the 
15 hard disk device 106 instead of drawing an illustration 
used as a query criteria from the beginning, and can 
make a similar image search using an image obtained by 
modifying the selected image. In this manner, upon 
drawing an illustration, neither drawing skills nor 
20 troublesome operations are required, and an efficient 
similar image search process can be executed. 

In the first embodiment, since a similar image 
search cannot be started unless an illustration is drawn 
on the user drawing area 21, icon image displayed on the 
25 areas 26a to 26h cannot be selected. However, this 
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shortcoming can be removed by combining another search 
method. 

For example, in an initial state or by user 
operation, icon images corresponding to image data 
5 randomly selected from those stored in the hard disk 
device 106 may be displayed on the areas 26a to 26h, 

On the other hand, the image search method of this 
embodiment may be combined with a conventional search 
method that uses attribute information such as a keyword 

10 or the like appended to an image as a query criteria. 
For example, when text "cat" is input, icon images 
corresponding to image data associated with "cat" are 
selected from those stored in the hard disk device 106, 
and are displayed on the areas 26a to 26h. The user 

15 selects a desired one of those icon images to copy it 
onto the user drawing area 21, and can make a similar 
image search using an image obtained by modifying the 
copied image as a query criteria. Also, attribute 
information such as a date of creation, management, 

20 correction, or the like of image data may be input as a 
query criteria. 
<Second Embodiment> 

In the first embodiment, image data stored in the 
hard disk device 106 are displayed, a desired one of the 

25 displayed images is selected, and an illustration used 
as a query criteria is drawn on the basis of the 
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selected image, thus realizing an efficient similar 
image search process. In the second embodiment, a 
desired color is selected from the displayed images to 
change the color used in drawing of an illustration, 
which is used as a query criteria, on the basis of the 
selected color, thus realizing an efficient similar 
image search process. 

Since the arrangement and control window of the 
image search apparatus are the same as those in the 
first embodiment, a detailed description thereof will be 
omitted. 

An outline of the processes executed by the image 
search apparatus of the second embodiment will be 
explained below with reference to Fig. 10, 

Fig. 10 is a flow chart showing an outline of the 
processes executed by the image search apparatus of the 
second embodiment. 

Note that the same step numbers denote the same 
steps as those in the flow chart in Fig. 3 of the first 
embodiment, and a detailed description thereof will be 
omitted. 

It is checked in step S295 if the user has clicked 
a point in the displayed icon image where a desired 
color is present while indicating that point with the 
cursor 22, If the user has clicked a given point (YES in 
step S295), the flow advances to step S296 to set the 
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color at the position of the cursor 22 upon clicking in 
step S295 to be the color used in drawing. On the other 
hand, if the user has not clicked any point (NO in step 
S295), the flow advances to step S297. 
5 It is checked in step S297 if search operation is 

to end. If search operation is not to end (NO in step 
S297), the flow returns to step S191. On the other hand, 
if search operation is to end {YES in step S297) , the 
processing ends. 

10 With the aforementioned processes, even when icon 

images themselves displayed as the search results on the 
areas 26a to 26h are not desired ones, if a portion of 
the icon image includes a color that the user wants to 
use in drawing of an illustration, the user moves the 

15 cursor 22 to the portion where the color is present, and 
clicks the mouse, thereby automatically setting that 
color as that used in drawing. Using the set color, the 
user can continue to draw the illustration on the user 
drawing area 21. 

20 The details of the processes executed in steps 

S295 to S297 will be described below. 
{Description of Step 3295} 

It is checked in step S295 if the user has clicked 
a point in the displayed icon image list while 

25 indicating that point by the cursor 22. If the user has 
not clicked any point, the flow advances to step S297. 
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If the user has clicked a given point, the flow advances 
to step S296. 

{Description of Step S296} 

In step S296, the color, which is being used in 
drawing of an illustration, is set at a pixel value at 
the clicked position. On the control window, the color 
of the cursor 22 is replaced by the set color, thus 
informing the user that the color of the cursor 22 has 
changed. Also, the color selection scroll bars 23 are 
changed to indicate the R, G, and B values of the set 
color. Normally, the color of the cursor 22 is changed 
when the user operates the color selection scroll bars 
23. However, in this case, the color selection scroll 
bars 23 are used to inform the user of the color set by 
the aforementioned method. The color set in this step is 
used to draw an illustration in step S191. 

Note that the pixel value used for setting the 
color may be obtained by copying image data displayed on 
the display unit 103 or image data corresponding to icon 
images displayed in a list in step S194. Depending on 
the display unit 103, both these image data may be equal 
to each other, but in general, the former data is easy 
to handle, and the latter data has higher precision. For 
example, when image data is compressed by JPEG and is 
saved in the hard disk device 106, each pixel value is 
expressed by 24 bits, but the display unit 103 often 
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makes a display while lowering the data precision to 8 
bits^ 15 bits, or 16 bits, 
{Description of Step S297} 

It is checked in step S2 97 if the processing end 
button 28 has been pressed. If the button 28 has been 
pressed, the processing ends. On the other hand, if the 
button 28 has not been pressed, the flow returns to step 
S191, and the user can continue to draw the illustration. 

As described above, according to the second 
embodiment, in order to set or change the color used in 
drawing of an illustration, the user need only move the 
cursor 22 to the position of a desired color in an image 
displayed on the window and click that position in place 
of making designation (using the color selection scroll 
bars 23) by means of R, G, and B values, which does not 
always come as an intuitive method for the user. In this 
manner, upon drawing an illustration, neither drawing 
skills nor troublesome operations are required, and an 
efficient similar image search process can be executed. 

In the second embodiment, the color used in 
drawing an illustration is set using icon images 
displayed as search results. However, another method may 
be used- For example, if a color can be selected from 
the illustration on the user drawing area 21, the moving 
amount of the cursor 22 can be reduced, thus further 
improving the work efficiency. 
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In a multi-task OS represented by Windows 
available from Microsoft Corp,, a plurality of 
applications can be started and can be simultaneously 
displayed on a single screen. When application software 
5 implemented by this embodiment, and another software, 

e.g., a WWW browser, photo-retouch software, or the like 
are simultaneously displayed to allow to use a color 
used in another software, the color selection range can 
be broadened, and the color used in drawing of an 

10 illustration can be selected more efficiently. 

On the other hand, when the pointing device 102a 
having a plurality of buttons including at least first 
and second buttons is used, a method of setting a color 
used in drawing an illustration can be selected 

15 according to user's purposes. For example, when the user 
has pressed the first button, the method of setting a 
color used in drawing an illustration described in the 
second embodiment is selected; when the user has pressed 
the second button, a method of setting a color used in 

20 drawing an illustration by another software is selected. 
These methods of setting a color used in drawing an 
illustration may be selected by forming dedicated 
buttons on the control window shown in Fig. 2 in place 
of using the buttons of the pointing device 102a. 

25 The color of the pixel value at the position 

indicated by the cursor 22, which is controlled by the 
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pointing device 102a, is used as a color used in drawing 
of an illustration. However, since an image normally 
contains noise, the color that the user requires does 
not always match the pixel value at that point. At this 
5 time, when the average value of the pixel values 
contained in a small region, e.g., a 3 X 3 (pixel) 
square region having the position indicated by the 
cursor 22 as the center is used as the color used in 
drawing of an illustration, the influences of noise can 

10 be suppressed. 

In the first and second embodiments, the mouse is 
used as the pointing device 102a, However, the present 
invention is not limited to such specific pointing 
device. For example, when a pen tablet that allows pen 

15 input is used, the operator can efficiently draw an 
illustration. Also, when a touch screen, which is 
integrated with the display unit 103 and allows the user 
to directly draw an illustration while observing the 
displayed window, is used, more intuitive drawing can be 

20 achieved. In addition, any other pointing devices may be 
used as long as they can input information to the 
computer system. 

In step S194, reduced-scale images corresponding 
to image data as the processing result of the similar 

25 image search process are displayed in a two-dimensional 
matrix. However, the reduced-scale images may be lined 
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up in a horizontal array (one-dimensionally) or may be 
three-dimensionally displayed in consideration of 
information in the depth direction. For example, images 
with higher similarity values may be displayed on the 
5 front side (to have larger image sizes) , and images with 
lower similarity values may be displayed on the far side 
(to have smaller image sizes) , thus achieving a list 
display which is easy to understand at a glance. 
<Third Embodiment> 

10 Upon drawing an illustration used as a query 

criteria, the illustration drawing method (first drawing 
method) described in the first embodiment, and that 
(second drawing method) described in the second 
embodiment may be selectively executed. For example, 

15 when the pointing device 102a having a plurality of 

buttons including at least first and second buttons is 
used, the drawing method can be selected according to 
user's purposes. For example, when the user has pressed 
the first button, the first drawing method is selected; 

20 when the user has pressed the second button, the second 
drawing method is selected. In place of using the 
buttons of the pointing device 102a, the first or second 
drawing method may be selected by forming dedicated 
buttons on the control window shown in Fig. 2. 
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Also, these drawing methods can be combined with 
the method of setting a color used in drawing of ^ an 
illustration. 

Note that the present invention may be applied to 
either a system constituted by a plurality of devices 
(e.g., a host computer, an interface device, a reader, a 
printer, and the like) , or an apparatus consisting of a 
single equipment (e.g., a copying machine, a facsimile 
apparatus, or the like) . 

The objects of the present invention are also 
achieved by supplying a storage medium, which records a 
program code of a software program that can implement 
the functions of the above-mentioned embodiments to the 
system or apparatus, and reading out and executing the 
program code stored in the storage medium by a computer 
(or a CPU or MPU) of the system or apparatus. 

In this case, the program code itself read out 
from the storage medium implements the functions of the 
above-mentioned embodiments, and the storage medium 
which stores the program code constitutes the present 
invention. 

As the storage medium for supplying the program 
code, for example, a floppy disk, hard disk, optical 
disk, magneto-optical disk, CD-ROM, CD-R, magnetic tape, 
nonvolatile memory card, ROM, and the like may be used. 

The functions of the above-mentioned embodiments 
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may be implemented not only by executing the readout 
program code by the computer but also by some or all of 
actual processing operations executed by an OS 
(operating system) running on the computer on the basis 
5 of an instruction of the program code. 

Furthermore, the functions of the above-mentioned 
embodiments may be implemented by some or all of actual 
processing operations executed by a CPU or the like 
arranged in a function extension board or a function 

10 extension unit, which is inserted in or connected to the 
computer, after the program code read out from the 
storage medium is written in a memory of the extension 
board or unit. 

When the present invention is applied to the 

15 storage medium, the storage medium stores program codes 

corresponding to the flow charts shown in Fig, 3, Fig. 4, 
and Figs. 6 to 10. 

As many apparently widely different embodiments of 
the present invention can be made without departing from 

20 the spirit and scope thereof, it is to be understood 
that the invention is not limited to the specific 
embodiments thereof except as defined in the appended 
claims . 
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WHAT IS CLAIMED IS: 

1. An image search apparatus for searching an image 
database that stores a plurality of image data for 
desired image data^. comprising: 

storage means for storing the plurality of image 
data in correspondence with image feature amounts of the 
image data; 

input means for inputting an image using an input 
window; 

image feature amount computing means for computing 
an image feature amount of the image input by said input 
means; 

image similarity computing means for computing 
image similarity on the basis of the image feature 
amount computed by said image feature amount computing 
means, and the image feature amounts of the image data 
stored in said storage means; 

image display means for displaying a list of image 
data as search results on the basis of the image 
similarity computed by said image similarity computing 
means; and 

read-out means for reading out the image data 
stored in said storage means into the input window. 

2. The apparatus according to claim 1, wherein said 
read-out means reads out one of the imag'e^ data displayed 
by said image display means into the input window. 
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3. The apparatus according to claim l,^,^wherein said 
read-out means comprises selection m^^ns for selecting 
desired image data from the image data displayed by said 
image display means, and 

5 said read-out means reads out the image data 

selected by said selection means into the input window. 

4. The apparatus according to claim^^J^^;^^ said 
image display means displays image data randomly 
selected from the image data stored in said storage 

10 means. 

5. The apparatus according to claim 1, ^herein said 
image display means displays designated-^'rmage data of 
the image data stored in said storage means. 

6. The apparatus according to claim 1, whe-rein said 
15 image display means comprises search mea^^for searching 

for desired image data by designating attribute 
information appended to the image data, and 

when a search is made by said search means, said 
image display means displays a list of image data found 

20 by the search of said search means. 

7- The apparatus according to claim 6,/wherein the 

attribute information is a keyword. y'' 
8. The apparatus according to claim 6, wherein the 

attribute information is at least one ot dates of 

25 creation, management, and modification of image data. 
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9. The apparatus according to claim 1, further 
comprising display control means for controlling to 
display the input window within a di^'lay window 
displayed by said image display means. 
5 10, An image search method for searching an image 
database that stores a plurality of image data for 
desired image data, comprising: 

the storage step of storing the plurality of image 
data in a storage medium in correspondence with image 
10 feature amounts of the image data; 

the image feature amount computing step of 
computing an image feature amount of an image input on 
an input window; 

the image similarity computing step of computing 
15 image similarity on the basis of the image feature 

amount computed in the image feature amount computing 
step^ and the image feature amounts of the image data 
stored in the storage medium in the storage step; 

the image display step of displaying a list of 
20 image data as search results on the basis of the image 
similarity computed in the image similarity computing 
step; and 

the read-out step of reading out the image data 
stored in the storage step into the input window. 
25 11. The method according to claim 10, wherein the 

read-out step includes the step of y^adlnq out one of 
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the image data displayed by said image display step into 
the input window. 

12. The method according to claim 10, wherein the 
read-out step comprises the selection' step of selecting 



5 desired image data from the image data displayed in the 
image display step, and 

the read-out step includes the step of reading out 
the image data selected in the selection step into the 
input window. 

10 13. The method according to claim 10, wherein the 

image display step includes the step of, displaying, as a 
default window, image data randomly /selected from the 
image data stored in the storage medium in the storage 
step, 

15 14. The method according to claim 10, ^wherein the 

image display step includes the step 'bf displaying, as a 
default window, designated image data of the image data 
stored in the storage medium in the storage step. 
15. The method according to claim 10/' wherein the 



20 image display step comprises the sea-^rch step of 
searching for desired image data by designating 
attribute information appended to the image data, and 

the image display step includes the step of 
displaying a list of image data found by a search in the 

25 search step, when the search is made in the search step. 




/ 
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16. The method according to claim 15, wherein the 
attribute information is a keyword.-' 

17. The method according to claim 15, wherein the 
attribute information is at least one pf'^dates of 

5 creation, management, and modification of image data. 

18. The method according to claim 10, further 
comprising the display control step ofx ^controlling to 
display the input window within a display window 
displayed in the image display step, 

10 19. A computer readable memory that stores a program 
code of an image search for searching an image database 
that stores a plurality of image data for desired image 
data, comprising: 

a program code of the storage step of storing the 

15 plurality of image data in a storage medium in 

correspondence with image feature amounts of the image 
data; 

a program code of the image feature amount 
computing step of computing an image feature amount of 
20 an image input on an input window; 

a program code of the image similarity computing 
step of computing image similarity on the basis of the 
image feature amount computed in the image feature 
amount computing step, and the image feature amounts of 
25 the image data stored in the storage medium in the 
storage step; 
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a program code of the image display step of 
displaying a list of image data as search results on the 
basis of the image similarity computed in the image 
similarity computing step; and 
5 a program code of the read-out step of reading out 

the image data stored in the storage step into the input 
window. 

20. An image search apparatus for searching an image 
database that stores a plurality of image data for 
10 desired image data^ comprising: 

storage means for storing the plurality of image 
data in correspondence with image feature amounts of the 
image data; ^ 
input means for inputting an image using an input 
15 window; 

image feature amount computing means for computing 
an image feature amount of the image input by said input 
means; 

image similarity computing means for computing 
20 image similarity on the basis of the image feature 

amount computed by said image feature amount computing 
means, and the image feature amounts of the image data 
stored in said storage means; 

image display means for displaying a list of image 
25 data as search results on the basis of the image 
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similarity computed by said image similarity computing 
means ; and 

designation means for designating a color used in 
the image to be drawn by said input means on the basis 
5 of the image data displayed by said image display means. 
21. The apparatus according to claim 20, wherein said 
designation means designates the color ^^ed in the image 
to be drawn by said input means by in'dicating one pixel 
in the image data displayed by said image display means. 

10 22. The apparatus according to claim 20, wherein said 
designation means designates the color used in the image 
to be drawn by said input means by indicating a 
predetermined region in the image data displayed by said 
image display means. 

15 23. The apparatus according to claim 22, wherein the 
color designated by indicating the pred^trermined region 
is an average value of pixel values contained in the 
small region . 

24. The apparatus according to claim 20, wherein said 
20 designation means designates the color u^'d^in the image 

to be drawn by said input means on the basis of image 
data to be displayed by said image display means. 

25. The apparatus according to claim 20, wherein said 
designation means designates the color u6ed in the image 

25 to be drawn by said input means on the basis of image 
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data which corresponds to image data displayed by said 
image display means ^ and is stored in said storage means, 

26. The apparatus according to claim 20, wherein said 
designation means designates the color used in the image 

5 to be drawn by said input means on the b^asis of an image 
drawn on a drawing area used by said input means. 

27. The apparatus according to claim 20, wherein said 
image feature amount computing means computes the image 
feature amount of the drawn image everyv-^£ime the image 

10 is modified by said input means. 

28. The apparatus according to claim 20, further 
comprising display control means for co^ntrplling to 
display the input window within a display window 
displayed by said image display means. 

15 29. An image search method for searching an image 
database that stores a plurality of image data for 
desired image data, comprising: 

the storage step of storing the plurality of image 
data in a storage medium in correspondence with image 
20 feature amounts of the image data; 

the image feature amount computing step of 
computing an image feature amount of an image input on 
an input window; 

the image similarity computing step of computing 
25 image similarity on the basis of the image feature 

amount computed in the image feature amount computing 
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step, and the image feature amounts of the image data 
stored in the storage medium in the storage step; 

the image display step of displaying a list of 
image data as search results on the basis of the image 
5 similarity computed in the image similarity computing 
step; and 

the designation step of designating a color used 
in the image to be drawn on the input window on the 
basis of the image data displayed in the image display 
10 step. 

30. The method according to claim 29/ wherein the 
designation step includes the step ^ designating the 
color used in the image to be drawn on the input window 
by indicating one pixel in the image data displayed by 

15 said image display step. 

31. The method according to claim 2 9, wherein the 
designation step includes the step iesignating the 
color used in the image to be drawn on the input window 
by indicating a predetermined region in the image data 

20 displayed in the image display step. 

32. The method according to claim 31, wherein the 
color designated by indicating the pred^r^ermined region 
is an average value of pixel values contained in the 
small region. 

25 33. The method according to claim 29, wherein the 

designation step includes the step of designating the 
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color used in the image to be drawn on the input window 
on the basis of image data to be displayed in the image 
display step. 

34. The method according to claim 29, wherein the 
5 designation step includes the step pf designating the 
color used in the image to be drawn on the input window 
on the basis of image data which corresponds to image 
data displayed in the image display step, and is stored 
in the storage medium in the storage step. 
10 35. The method according to claim 29, wherein the 

designation step includes the step of designating the 



on the basis of an image drawn on a user drawing area, 
36, The method according to claim 29, wherein the 



15 image feature amount computing step includes the step of 
computing the image feature amount of the drawn image 
every time the image input on the input window is 
modified. 

37. The method according to claim 29, further 

20 comprising the display control step of /Controlling to 
display the input window within a display window 
displayed in the image display step. 

38. A computer readable memory that stores a program 



code of an image search for searching an image database 
25 that stores a plurality of image data for desired image 
data, comprising: 



color used in the image to be drawr>/at said input step 
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a program code of the storage step of storing the 
plurality of image data in a storage medium in 
correspondence with image feature amounts of the image 
data; 

5 a program code of the image feature amount 

computing step of computing an image feature amount of 
an image input on an input window; 

a program code of the image similarity computing 
step of computing image similarity on the basis of the 
10 image feature amount computed in the image feature 

amount computing step, and the image feature amounts of 
the image data stored in the storage medium in the 
storage step; 

a program code of the image display step of 
15 displaying a list of image data as search results on the 
basis of the image similarity computed in the image 
similarity computing step; and 

a program code of the designation step of 
designating a color used in the image to be drawn on the 
20 input window on the basis of the image data displayed in 
the image display step. 

39. An image search apparatus for searching an image 
database that stores a plurality of image data for 
desired image data, comprising: 
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storage means for storing the plurality of image 
data in correspondence with image feature amounts of the 
image data; 

selection means for selecting one of a plurality 
5 of different input methods; 

input means for inputting an image using the input 
method selected by said selection means; 

image feature amount computing means for computing 
an image feature amount of the image input by said input 
10 means; 

image similarity computing means for computing 
image similarity on the basis of the image feature 
amount computed by said image feature amount computing 
means, and the image feature amounts of the image data 
15 stored in said storage means; and 

image display means for displaying a list of image 
data as search results on the basis of the image 
similarity computed by said image similarity computing 
means . 

20 40. The apparatus according to claim 39, wherein the 
input methods include at least a f irsj^-dnput method of 
inputting an image on the basis of image data stored in 
said storage means, and a second input method of 
inputting an image on the basis of a color of image data 

25 stored in said storage means. 



- 50 - 



41- The apparatus according to claim 39, wherein the 
input methods include at least a third inpyt method of 
inputting an image on the basis of im^^e" data stored in 
said storage means, and a fourth input method of 
5 inputting an image on the basis of a color managed by 
another application . 

42, The apparatus according to claim 39, wherein said 
selection means comprises a pointing devi_c^ a 
plurality of buttons, and 

10 the plurality of different input methods are 

assigned to the plurality of buttons, and one of the 
plurality of different input methods is selected by 
selecting one of the plurality of buttons. 

43. The apparatus according to claim 39, wherein said 
15 selection means comprises a plurality o^^.-^uttons formed 

on an input window displayed by said image display means, 
and 

the plurality of different input methods are 
assigned to the plurality of buttons, and one of the 
20 plurality of different input methods is selected by 
selecting one of the plurality of buttons. 
44- An image search method for searching an image 
database that stores a plurality of image data for 
desired image data, comprising: 
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the storage step of storing the plurality of image 
data in a storage medium in correspondence with image 
feature amounts of the image data; 

the selection step of selecting one of a plurality 
5 of different input methods; 

the input step of inputting an image using the 
input method selected in the selection step; 

the image feature amount computing step of 
computing an image feature amount of an image input in 
10 the input step; 

the image similarity computing step of computing 
image similarity on the basis of the image feature 
amount computed in the image feature amount computing 
step, and the image feature amounts of the image data 
15 stored in the storage medium in the storage step; and 

the image display step of displaying a list of 
image data as search results on the basis of the image 
similarity computed in the image similarity computing 
step. 

20 45- The method according to claim 44, wherein the 

input methods include at least a f irs€" input method of 
inputting an image on the basis of image data stored in 
the storage medium in the storage step, and a second 
input method of inputting an image on the basis of a 

2 5 color of image data stored in the storage medium in the 
storage step. 
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46. The method according to claim 44, wherein the 
input methods include at least a thirtf'" input method of 
inputting an image on the basis of image data stored in 
the storage medium in the storage step, and a fourth 

5 input method of inputting an image on the basis of a 
color managed by another application, 

47, The method according to claim 44, wherein the 
plurality of different input methods^'are^respectively 
assigned to a plurality of buttons provided to a 

10 pointing device, and the selection step includes the 
step of selecting one of the plurality of different 
input methods on the basis of a selection state of the 
plurality of buttons. 



15 plurality of different input methods are respectively 
assigned to a plurality of buttons formed on an input 
window displayed in the image display step, and the 
selection step includes the step of selecting one of the 
plurality of different input methods on the basis of a 

20 selection state of the plurality of buttons. 

49. A computer readable memory that stores a program 
code of an image search for searching an image database 
that stores a plurality of image data for desired image 
data, comprising: 

25 a program code of the storage step of storing the 

plurality of image data in a storage medium in 



48. 



The method according to claim 




wherein the 
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correspondence with image feature amounts of the image 
data; 

a program code of the selection step of selecting 
one of a plurality of different input methods; 

a program code of the input step of inputting an 
image using the input method selected in the selection 
step; 

a program code of the image feature amount 
computing step of computing an image feature amount of 
an image input in the input step; 

a program code of the image similarity computing 
step of computing image similarity on the basis of the 
image feature amount computed in the image feature 
amount computing step, and the image feature amounts of 
the image data stored in the storage medium in the 
storage step; and 

a program code of the image display step of 
displaying a list of image data as search results on the 
basis of the image similarity computed in the image 
similarity computing step. 
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ABSTRACT OF THE DISCLOSURE 
A plurality of image data are stored in a hard 
disk device in correspondence with their image feature 
amounts. An image is input using an input window. A CPU 
computes the image feature amount of the input image. 
The CPU also computes image similarity on the basis of 
the computed image feature amount and those of image 
data stored in the hard disk device, A list of image 
data as search results are displayed on a display unit 
on the basis of the computed image similarity. Image 
data stored in the hard disk device are read out into 
the input window. 



- 55 - 



1/10 







CO 

-o 




o 




CD 




LO 

-O 




2/10 







o 

CD 
CM 



d 

UL 




CD 
C\J 





CD 
CD 



3/10 



S191 



C 



START 



3 



FIG. 3 



DISPLAY IMAGE INTERACTIVELY 



S192 



S193 



S194 



S195 



COMPUTE IMAGE FEATURE 
AMOUNT 



SEARCH SIMILAR IMAGE 




S196 



COPY ICON TO DRAWING AREA 



S197 




4/10 



FIG. 4 




S35 



XO <r- X1 

yo ^ yi 




5/10 



X 



m 
if* 

m 
□ 



d 

LL 



(2.0) 




p 
1 — 


T— 
1 — 


(0.0) 


T— 



S51 



6/10 



C 



START 



J 



FIG. 6 



S52 



j^O 



S53 



t 



^0 



S54 



dfk) ^ R AVERAGE VALUE OF AREA (i, j) 
d(k + 1) ^ G AVERAGE VALUE OF AREA (i, j) 
d(|< + 2) ^ B AVERAGE VALUE OF AREA (i. j) 



S55 



k + 3 



i 



S56 



i <— i + 1 




c 



7/10 



START 



1 



3 



FIG. 



S61 



DR ^0 
DG <-0 
DB ^0 



S62 



Y ^ YO 



S63' 



S64- 



X^XO 








< 1 


DR ^ DR + R(X,Y) 
DG ^ DG + G(X,Y^ 
DB ^ DB + B(X,Y) 






S68 




S69- 



DR ^ DR / ((X1 


- X0)*(Y1 


- YO)) 


DG ^ DG / {(X1 


- X0)*(Y1 


- YO)) 


DB ^ DB / ((X1 


- X0)*(Y1 


- YO)) 



8/10 

FIG. 8 



S71 



S72 



S73 



S74 



S76 




S81 



S810 




10/10 



FIG. 10 




S297 



END OF 
SEARCH OPERATION? 



YES 



COMBINED DECLARATION AND POWER OF ATTORNEY 
FOR PATENT APPLICATION 

(Pagel) 



As a below named inventor, I hereby declare that: 



My residence, post office address and citizenship are as stated below next to my name. 



I believe I am the original, first and sole inventor (if only one name is listed below) or an original, first and joint inventor (if plural 

names are listed below) of the subject matter which is claimed and for which a patent is sought on the invention entitled 

IMAGE SEARCH APPARATUS AND METHOD, AND COMPUTER READABLE 

MEMORY , 

the specification of which | X | is attached hereto | | was filed on as United States 

Application No. or PCX International Application No. 

and was amended on (if applicable). 



I hereby state that I have reviewed and understand the contents of the above-identified specification, including the claims, as amended 
by any amendment referred to above. 

I acknowledge the duty to disclose information which is material to patentability as defined in 37 CFR § 1 .56. 

I hereby claim foreign priority beneiits under 35 U.S.C. § 119(a)-(d) or §3 65(b), of any foreign application (s) for patent or inventor's 
certificate, or § 365(a) of any PCX international application which designates at least one country other than the United States, listed below 
and have also identified below any foreign application for patent or inventor's certificate, or PCX international application having a filing date 
before that of the application on which priority is claimed: 

(Yes/No) 

Country Application No. Filed (Day/Mo. A^r.) Prionty Claimed 

Japan 10-244586 August 3 1 , 1 998 Yes 

Japan 1 0-244588 August 3 1 , 1 998 Yes 

Japan 11-214265 July 28, 1999 Yes 

I hereby claim the benefit under 35 U.S.C. § 120 of any United States appHcation(s), or § 365(c) of any PCX international application 
designating the United States, listed below and, insofar as the subject matter of each of the claims of this application is not disclosed in the 
prior United States or PCX international application in the manner provided by the first paragraph of 35 U.S.C. § 112, 1 acknowledge the duty 
to disclose information which is material to patentability as defined in 37 C.F.R. § 1 .56 which became available between the filing date of the 
prior application and the national or PCX international filing date of this application. 

Application No. Filed (Day/Mo. /Yr.) Status (Patented, Pending, Abandoned) 



N/A 



I hereby appoint the practitioners associated with the firm and Customer Number provided below to prosecute this application and 
to transact all business in the Patent and Xrademark Office connected therewith, and direct that all correspondence be addressed to the address 
associated with that Customer Number: 



FITZPATRICK, CELLA, HARPER & SCINTO 
Customer Number: 05514 

I hereby declare that all statements made herein of my own knowledge are true and that all statements made on information and belief 
are believed to be true; and further that these statements were made with the knowledge that willful false statements and the like so made are 
punishable by fine or imprisonment, or both, under Section 1001 of Xitle 18 of the United States Code and that such willful false statements 
may jeopardize the validity of the application or any patent issued thereon. 

Full Name of Sole or First Inventor KUNIHIRQ YAMAMQTO 

Inventor's signature 

Date Citizen/Subject of JAPAN 

Residence 36-30-505, TsuruTtiichuo 4-chome, Tsurumi-ku, 

Yokohama - shi , Kanacrawa - ken , Japan 

Post Office Address c/o Canon Kabushiki Kaisha, 

30-2, Shimomaruko 3-chome, Qhta-ku, Tokyo, Japan 



COMBINED DECLARATION AND POWER OF ATTORNEY 
FOR PATENT APPLICATION 

(Page 2) 



Full Name of Second Joint Inventor, if any KENTARO MATSUMQTQ 

Second Inventor's signature 

Date Citizen/Subject of JAPAN 

Residence 19-2, Hikawadai 1-chome, Hiaashikurume-shi . Tokyo, Japan 



Post Office Address c/o Canon Kabushiki Kaisha, 

3 0-2, Shimomaruko 3 - chome , Qhta-ku , Tokyo , Japan 

Full Name of Third Joint Inventor, if any KIYOSHI KUSAMA 

Third Inventor's signature 

Date Citizen/Subject of JAPAN 

Residence c/o Canon Daini Moecfino Ryo, 27-6, Moeaino. Aoba-ku, 

Yokohama - shi , Kanagawa - ken , Japan 

Post Office Address c/o Canon Kabushiki Kaisha, 

30-2, Shimomaruko 3-chome, Qhta-ku, Tokyo, Japan 

Full Name of Fourth Joint Inventor, if any 

Fourth Inventor's signature 

Date Citizen/Subject of 

Residence 



Post Office Address 



Full Name of Fifth Joint Inventor, if any 

Fifth Inventor's signature 

Date Citizen/Subject of 

Residence 



Post Office Address 



Full Name of Sixth Joint Inventor, if any 

Sixth Inventor's signature 

Date Citizen/Subject of. 

Residence 



Post Office Address 



NY_MAIN 24532 v 1 



